The importance of reporting and interpreting critical effect sizes

Ambra Perugini, Filippo Gambarota, Enrico Toffalini, Daniël Lakens, Massimiliano Pastore, Livio Finos, Psicostat, Gianmarco Altoè

 Meta-Rep 29 - 31 October 2024 | Munich

Where are we?

  • In the Psychological Sciences field
  • Obvious but not so obvious …
  • There is lack of awareness of the link between statistical significance, effect size and sample size

Let’s start with an example

  • We did not find a significant correlation between variable X and variable Y (\(n = 18\) , \(r = .40\), \(p = .100\))

  • We found a significant correlation between variable X and variable Y (\(n = 1000\), \(r = .08\), \(p = .011\))

What is a critical effect size?

It is the smallest effect size associated with a chosen \(\alpha\) (therefore significant), given sample size, test chosen and direction of the hypothesis.

It would be ideal to plan \(n\) to reach a power of 80%. Often we are in a scenario where it was not possible to pre-plan sample size, either for limited resources or because we accessed a large database.

  • In the first case optimal power cannot be reached

  • In the second case even small effects reach statistical significance

  • In the first example the correlation was not significant, but the critical effect size was \(r = ±.468\). Are we sure that a correlation of .30 is not relevant?

  • In the second example the critical effect size was \(r = ± 0.062\), depending on the construct under investigation such a small effect might not have practical meaning.

How do critical effect sizes help us?

Before conducting the study:

  • they highlight the importance of not merely focusing on \(p < .05\) when interpreting results.

In front of the dead corpse:

  • They help researchers and reviewers to better contextualize research findings.

And more..

  • From a didactic point of view they help comprehend the relationship between statistical significance, sample size and effect size.

How to compute critical effect sizes


Our package helps to compute critical effect sizes for correlations, group comparisons, linear regressions and meta-analysis.

library(criticalESvalue)
n <- 30
critical_t2sp(n = n, conf.level = 0.95, hypothesis = "two.sided")$dc
[1] 0.528076

What next?

  • Reality is always more complex than a correlation of a linear model

  • We will work on the implementation of critical effect sizes for more complex models (i.e. Structural Equation Modeling)

To wrap up

  • Of course it would be better to pre-plan sample size by formalizing a plausible effect size BEFORE conducting the study

  • But.. when it is not possible to plan \(a\) \(priori\) the study, critical effect sizes allow to undertand beforehand the limitations of the study design, without having to specify a plausible effect.

  • Our motto is TBT:

Thinking Before Testing!

To find out more:


Here you can find our package, examples on how to use the functions and the draft of the paper:


https://github.com/psicostat/criticalESvalue


Ambra Perugini

ambra.perugini\(@\)phd.unipd.it

https://psicostat.dpss.psy.unipd.it/people.html